Apparatus, system and method for projecting images onto predefined portions of objects

ABSTRACT

Apparatus, systems and methods are provided for illuminating objects in a projection area. The system includes a computing device, a projector and a camera. The computing device stores a digital model of an object, and illumination data having lighting parameters and a reference to the digital model. The projector, or another light source, projects structured light onto the projection area, and the camera simultaneously captures an image of the projection area. The computing device receives the captured image, determines a position and orientation of the object by comparing the digital model to the captured image, and then generates a canvas image including a region matching the determined position and orientation of the object. The projector projects the canvas image onto the projection area. A predefined portion of the object corresponding to the reference in the illumination data is thereby illuminated according to the lighting parameters.

FIELD

The specification relates generally to digital projection systems, andspecifically to a method, system and apparatus for projecting digitalimages onto a projection area, and for illuminating the projection area,and particularly predefined portions of objects in the projection area,adapting for relative movement between the projection system and theobjects.

BACKGROUND

Projection systems are typically used to project fixed, predeterminedimages onto flat, static surfaces. When the surface being projected onis not flat, careful calibration may be required to account for theshape of the surface. Projecting images onto moving objects presentsadditional complications: moving the projector to track the motion of anobject, as with a mobile spotlight, can be cumbersome. When multiplemoving objects are involved, multiple projectors are also required, eachof which may be required to move independently of the others to trackits assigned object.

In addition to the obstacles involved in enabling projection equipmentto follow moving objects, the images projected onto those moving objectsmay still appear distorted. Relative motion between projectors andtarget objects therefore renders accurate projection difficult toachieve.

SUMMARY

According to an aspect of the specification, a method of illuminatingobjects in a projection area is provided, comprising: storing, in amemory of a computing device: geometry data defining a digital model ofan object; and illumination data containing a record having (i) one ormore parameters defining characteristics of light to be projected ontothe object, and (ii) a reference to the digital model; controlling alight source connected to the computing device to project structuredlight onto the projection area; controlling a camera connected to thecomputing device to capture an image of the projection area during theprojection of structured light; receiving the captured image at thecomputing device from the camera and determining a position andorientation of the object in the projection area by comparing thegeometry data to the captured image; generating a canvas image at thecomputing device, including a region matching the determined positionand orientation of the object, the region having a fill defined by theone or more parameters; transmitting the canvas image to a projectorconnected to the computing device, for projection onto the projectionarea, whereby a portion of the object corresponding to the reference inthe illumination data is illuminated according to the one or moreparameters.

According to another aspect of the specification, a computing deviceconfigured to perform the method, and a non-transitory computer-readablemedium containing instructions for execution on the computing device,are provided.

According to a further aspect of the specification, a system includingthe computing device, a light source, and the camera are provided.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Embodiments are described with reference to the following figures, inwhich:

FIG. 1 depicts a system for projecting digital images, according to anon-limiting embodiment;

FIG. 2 depicts geometry data maintained in the system of FIG. 1,according to a non-limiting embodiment;

FIG. 3 depicts image data maintained in the system of FIG. 1, accordingto a non-limiting embodiment;

FIG. 4 depicts a method of projecting digital images, according to anon-limiting embodiment;

FIG. 5 depicts a method of performing block 410 of the method of FIG. 4,according to a non-limiting embodiment;

FIG. 6 depicts the projection of structured light as part of the methodof FIG. 5, according to a non-limiting embodiment;

FIG. 7 depicts an image captured by performing the method of FIG. 5,according to a non-limiting embodiment;

FIG. 8 depicts the results of the performance of block 525 of the methodof FIG. 5, according to a non-limiting embodiment;

FIG. 9 depicts a canvas image generated at block 415 of the method ofFIG. 4, according to a non-limiting embodiment;

FIG. 10 depicts the performance of block 420 of the method of FIG. 4,according to a non-limiting embodiment;

FIG. 11 depicts another performance of block 420 of the method of FIG.4, according to a non-limiting embodiment;

FIG. 12 depicts another performance of the methods of FIGS. 4 and 5following movement of the camera and projector of FIG. 1;

FIG. 13 depicts another example of structured light used in the methodof FIG. 5, according to a non-limiting embodiment;

FIG. 14 depicts illumination data maintained in the system of FIG. 1,according to another non-limiting embodiment;

FIG. 15 depicts a method of illuminating objects, according to anon-limiting embodiment;

FIG. 16 depicts a canvas image generated at block 415 a of the method ofFIG. 15, according to a non-limiting embodiment; and

FIG. 17 depicts the performance of block 420 a of the method of FIG. 15,according to a non-limiting embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1 depicts a projection system 100 for projecting digital images.System 100 includes one or more projectors, such as a projector 104,arranged to project (illustrated by the dashed lines extending fromprojector 104) digital images received from a computing device 108 ontoa projection area 112. System 100 also includes one or more cameras,such as a camera 116 for capturing images of projection area 112 andsending the captured images to computing device 108. Projector 104 andcamera 116 are mounted at known, preferably fixed, positions relative toone another, such that the position of any object in relation to camera116 can be used to determine the position of that object in relation toprojector 104. In addition, the field of view of camera 116 encompassesthe entirety of projection area 112; that is, camera 116 is configuredto capture images of an area at least as large as the area onto whichprojector 104 projects light.

As will be discussed in detail herein, projector 104 and computingdevice 108, in conjunction with camera 116, are configured to projectthe above-mentioned digital images in such a way that predeterminedportions of the images are projected onto one or more objects, such asan object 120 (in the example of FIG. 1, a human subject) in projectionarea 112. Further, the system is configured to project the images suchthat those predetermined portions of the images track any relativemotion between object 120 and projector 104. As a result, thepredetermined portions remain substantially static in relation to object120, even as object 120 moves within projection area 112. In otherwords, the predetermined portions of the images are given the appearanceof being affixed to specific portions of object 120.

Before discussing the operation of system 100 in detail, the componentsof system 100 will be described further.

Projector 104 can be any suitable type of projector, or combination ofprojectors. Projector 104 is stationary in the present example, but canbe mobile in other embodiments. Projector 104 thus includes one or morelight sources, one or more modulating elements for modulating light fromthe light sources to produce a desired image provided by computingdevice 108, and a lens assembly for directing the modulated light ontoprojection area 112. In some examples, projector 104 can project imagesusing light falling within the spectrum visible to the human eye (thatis, wavelengths of about 390 to 700 nm), outside the visible spectrum(for example, infrared light having a wavelength greater than about 750nm), or both simultaneously.

Camera 116 can be any suitable type of digital camera, and thus includesa lens assembly for focusing reflected light incident on camera 116 fromprojection area 112. Camera 116 also includes an image sensor onto whichthe incident light is focused by the lens assembly. Camera 116 isconfigured to transmit the image data produced by the image sensor tocomputing device 108.

Computing device 108 can be based on any suitable server or personalcomputer environment. In the present example, computing device 108 is adesktop computer housing one or more processors, referred to genericallyas a processor 124, The nature of processor 124 is not particularlylimited. For example, processor 124 can include one or more generalpurpose central processing units (CPUs), and can also include one ormore graphics processing units (GPUs). The performance of the variousprocessing tasks discussed herein can be shared between the CPUs andGPUs, as will be apparent to a person skilled in the art.

Processor 124 is interconnected with a non-transitory computer readablestorage medium such as a memory 126. Memory 126 can be any suitablecombination of volatile (e.g. Random Access Memory (“RAM”)) andnon-volatile (e.g. read only memory (“ROM”), Electrically ErasableProgrammable Read Only Memory (“EEPROM”), flash memory, magneticcomputer storage device, or optical disc) memory. In the presentexample, memory 112 includes both a volatile memory and a non-volatilememory.

Computing device 108 can also include one or more input devices 128interconnected with processor 124, such as any suitable combination of akeyboard, a mouse, a microphone, and the like. Such input devices areconfigured to receive input and provide data representative of suchinput to processor 108. For example, a keyboard can receive input from auser in the form of the depression of one or more keys, and provide dataidentifying the depressed key or keys to processor 124.

Computing device 108 further includes one or more output devicesinterconnected with processor 124, such as a display 130 (e.g. a LiquidCrystal Display (LCD), a plasma display, an Organic Light Emitting Diode(OLED) display, a Cathode Ray Tube (CRT) display). Other output devices,such as speakers (not shown), can also be present. Processor 124 isconfigured to control display 130 to present images to a user ofcomputing device 108.

Computing device 108 also includes a data interface 132 interconnectedwith processor 124, for carrying data from processor 124 to projector104, and for carrying data from camera 116 to processor 124. The natureof interface 132 is not particularly limited. In general, interface 132includes the necessary hardware elements to enable communicationsbetween computing device 108 and projector 104 and camera 116. Interface132 can also include multiple interfaces, for example if differentcommunication technologies are used by projector 104 and camera 116.

Computing device 108 is configured to perform various functions, to bedescribed herein, via the execution by processor 124 of applicationsconsisting of computer readable instructions maintained in memory 126.Specifically, memory 126 stores an application 134 includingcomputer-readable instructions executable by processor 124. Whenprocessor 124 executes the instructions of application 134, processor124 is configured to perform various functions in conjunction with theother components of computing device 108, and with projector 104 andcamera 116. Processor 124 is therefore described herein as beingconfigured to perform those functions via execution of application 134.In the discussion below, when computing device 108 generally, orprocessor 124 specifically, are said to be configured to perform acertain action or function, it will be understood that the performanceof the action or function is caused by the execution of application 134by processor 124.

Memory 126 also stores geometry data 136 identifying geometricalfeatures of various objects that can appear in projection area 112, andimage data 138 defining one or more images that can be projected ontoprojection area 112 (by projector 104, under the control of computingdevice 108).

Turning to FIG. 2, an example of geometry data 136 is provided.Specifically, geometry data 136 in the present example identifiesgeometrical features of object 120 (the human subject) shown in FIG. 1.In other words, geometry data 136 defines a digital model of object 120.The model is preferably in three dimensions, but in other examples,geometry data 136 can be simplified by defining a model in only twodimensions, even for a three dimensional object. It will now be apparentthat geometry data 136 does not define a model for another object,object 122 (a short wall placed in projection area 112, as seen in FIG.1). In the present example, this is because object 122 is assumed to notbe an object “of interest” (that is, an object onto which digital imagesare to be projected). In general, geometry data 136 contains predefinedmodels for any objects onto which digital images are to be projected,but does not define models for other objects.

FIG. 2 shows a three dimensional model 200 of object 120, as defined bygeometry data 136. FIG. 2 also shows an example of geometry data 136itself. As seen in FIG. 2, geometry data 136 includes a plurality ofrecords each identifying features of a certain part of object 120. Inthe present example, where object 120 is a human subject, geometry data136 includes one record for each of the following parts: a torso 204(modeled as a rectangular prism having a height of 50 cm, a width of 30cm, and a depth of 20 cm); left and right arms 208 and 212 (each modeledas a cylinder having a length of 75 cm and a radius of 5 cm); left andright legs 216 and 220 (each modeled as a cylinder having a length of100 cm and a radius of 10 cm); and a head 224 (modeled as a spherehaving a radius of 12 cm). The above list of parts, and theirdimensions, are provided solely for illustrative purposes—a wide varietyof parts and dimensions can be provided in geometry data 136.

Geometry data 136 also defines the connections between various parts.For example, as seen in the “connections” column of geometry data 136,all the parts other than torso 204 are marked as being connected totorso 204. Although not shown in FIG. 2, geometry data 136 can includecoordinates or other identifications of where the connections betweenparts are. Geometry data 136 can also include a wide variety of otherdata, such as permissible ranges of motion of various parts relative toother parts, and the like. Model 200 of object 120 is simplified forillustrative purposes, but it is contemplated that geometry data 136 candefine a more complex model of object 120, including individual segmentsof limbs, and including more detailed shape data than the prisms shownin FIG. 2. For example, geometry data 136 may define a wire-frame modeldivided into any number of parts, each part including a plurality ofsurfaces joined by edges.

It is also contemplated that any suitable format can be used to storegeometry data 136. That is, although geometry data 136 is shown in atabular format in FIG. 2, a wide variety of other formats can also beused. The technology used to acquire or create geometry data 136 is alsonot particularly limited. For example, any suitable computer aideddesign (CAD) software can be used to generate geometry data. In otherexamples, geometry data 136 can be generated by scanning object 120itself, using depth-mapping, LIDAR, multi-camera 3D imaging, light fieldcamera imaging, structured light imaging, or other suitabletechnologies.

Turning now to FIG. 3, an example of image data 138 will be discussed.Image data 138, in the present example, defines two images, 300 and 304.Each image is stored in memory 126 as an image file. The image filesthat constitute image data 138 each include various types of data, whichcan be organized in fields or other suitable data structures. The typesof data can include a file name, used by computing device 108 toidentify and retrieve the image; metadata, such as an image size, dateof creation (not shown) and the like; colour and brightness data; and areference to geometry data 136, also referred to as mapping metadata.

Thus, image 300 is defined by a file including a file name field 306(“stop.bmp”) and an image size field 308 specifying the size of image300 (two hundred by two hundred pixels). It is contemplated that imagesize field 308 can be omitted, or can be presented in a different manner(for example, rather than or in addition to pixel-based size, one orboth of an aspect ratio and a total number of pixels can be specified).Image 300 is also defined by colour and brightness data 310 in the formof a pixel array. Each pixel in the array includes three colour values(one for each of a red channel “R”, a green channel “G”, and a bluechannel “B”) and one brightness value (“L”). In the present example, thepixel array 310 includes forty thousand pixels (not all shown),corresponding to the dimensions specified in field 308. A wide varietyof types of colour and brightness data are contemplated. For example,image 300 may be defined by vector data rather than by individual pixelvalues. In some examples, even when pixel arrays are used, compressionalgorithms may be used to reduce the number of individual pixels whichmust be defined in the image file. In addition, red, green, blue andbrightness values can be substituted by any other suitable colour model(e.g. CMYK).

Further, the file defining image 300 includes a mapping metadata field312, which refers to a part of model 200 as discussed earlier. Mappingmetadata field 312 is used to determine the placement of image 300 onobject 120 when image 300 is projected onto projection area 112 byprojector 104. Mapping metadata field 312 can also include additionaldata specifying the orientation of image 300 with respect to torso 204,as well as the exact location on torso 204 of image 300 (for example, byway of coordinates or distances from specified edges of torso 204).

Image 304 is defined by a file having analogous components to thosediscussed above. Thus, the file defining image 304 includes a file namefield 316, a size field 318, a pixel array 320, and a mapping metadatafield 322.

It is contemplated that in some examples, individual pixels or blocks ofpixels (in raster, or bitmap, images as shown in FIG. 3) or vector data(in vector images, not shown) can include mapping metadata, rather thanmapping metadata being confined to a separate field from the pixelarrays 310 and 320. That is, certain pixels can be mapped to aparticular portion of geometry data 136. In general, therefore, imagedata 138 includes mapping metadata referring to at least a portion ofgeometry data 136. In addition, although the example image data in FIG.3 defines two static images, image data 138 can also define a motionpicture in the form of a sequence of images or a video stream.

Having described the components of system 100, the operation of system100 will now be described in detail, with reference to FIG. 4. FIG. 4depicts a method 400 of projecting images onto projection area 112. Theperformance of method 400 will be discussed in conjunction with itsperformance in system 100, although it is contemplated that method 400can also be performed by other suitable systems.

Method 400 begins at block 405, at which computing device 108 isconfigured to store geometry data 136 and image data 138, as discussedabove. It is contemplated that geometry data can be stored for a widevariety of objects, including object 120, onto which images are to beprojected when such objects are present in projection area 112. Further,it is contemplated that image data can be stored for a wide variety ofimages, which can refer to any combination of the objects described bygeometry data 136. In addition, as mentioned earlier image data 138 canalso specify a sequence for the various images defined therein, forexample when a slideshow or video is to be projected onto projectionarea 112. In such cases, each image file can include a sequence number,or image data 138 can include a video file including several imagesub-files in a particular sequence. For the present example performanceof method 400, however, geometry data 136 and image data 138 are assumedto be as described earlier herein (that is, defining a single object andtwo images).

Proceeding to block 410, computing device 108 is configured to detectany objects within projection area 112 that correspond to geometry data136, and to determine the position and orientation of each detectedobject relative to projector 104. A variety of methods for identifyingand determining the position and orientation of objects withinprojection area 112 are contemplated. For example, a depth mappingapparatus (not shown), such as a LIDAR apparatus, can be connected tocomputing device 108 and can generate a depth map of projection area112. Computing device 108 can then determine whether any objectsdescribed by geometry data 136 are present in the depth map. Otherrange-finding and depth-mapping apparatuses can also be implemented.

In other examples, such range-finding or depth mapping technologies canbe replaced by, or supplemented with, location-finding technologies suchas a GPS receiver (not shown) affixed to object 120 which determines itslocation and transmits the location to computing device 108.

In the present example, the performance of block 410 involves bothprojector 104 and camera 116, as will be discussed in connection withFIG. 5. FIG. 5 depicts an example performance of block 410 of method400. Beginning at block 500, computing device 108 is configured tocontrol projector 104, or another light source separate from projector104 (not shown), to project a structured light pattern onto projectionarea 112. The nature of the structured light is not particularlylimited. Structured light projections can include an arrangement ofparallel lines of light, grids of light, Gray-code patterns, full framesof light, and the like. In general, the structured light projected atblock 500 is selected such that portions of the structured light thatare reflected back from projection area 112 can be captured by camera116 and used by computing device 108 to determine object positions andorientations, as will be discussed below.

An example of structured light 600 is shown in FIG. 6 as a series 600 ofparallel lines of light projected onto projection area 112. As seen inFIG. 6, objects 120 and 122 (and indeed, any other object in projectionarea 112) obstruct portions of structured light 600. The parallel linesshown in FIG. 6 are provided simply for the purposes of illustration—anysuitable pattern of structured light can be used at block 500.

In the present example, structured light 600 is projected by projector104 itself, as projector 104 is well suited to generating structuredlight 600 using the same light modulation technology as is used forprojector 104's primary purpose of projecting digital images. Structuredlight 600 can be either visible or invisible light (that is, within thespectrum visible by human observers, or outside the visible spectrum).As mentioned earlier, projector 104 can therefore be capable ofprojecting both visible and invisible light; an example of such aprojector is provided in US Published Patent Application No,2010/0110308. When structured light 600 is within the visible spectrum,it can nevertheless be made invisible to human observers by beingprojected at block 500 for a time period sufficiently short as to beimperceptible to observers. As demonstrated in the practice ofsubliminal messaging in motion pictures, when the duration of thestructured light intervals are sufficiently short, they are below thethreshold of conscious perception by humans.

Returning to FIG. 5, at block 505 computing device 108 is configured tocontrol camera 116 to capture an image of projection area 112. Whenstructured light 600 is projected for only a certain time interval (asin the scenario using visible light described above), the shutter ofcamera 116 is synchronized to open only during the interval for whichstructured light 600 is projected. Synchronization can be provided by awired or wireless signal from computing device 108, such that the imageis captured by camera 116 at block 505 during the projection ofstructured light 600 on projection area 112. As a result, the effect onthe captured image of any ambient light and of any projected imagery onprojection area 112 is minimized, and the captured image includes anyportions of structured light 600 reflected from projection area 112 backtowards camera 116. The shutter speed with which camera 116 captures theimage at block 505 is not particularly limited, but is sufficiently highto avoid motion blurring in the captured image due to movement of object120. The image captured at block 505 is provided to computing device 108for storage in memory 126, for further processing.

The image captured by camera 116 and sent to computing device 108 atblock 505 is shown in FIG. 7 as image 700. Image 700 includes signatures704 which are reflected portions of structured light 600 that areindicative of obstructions to structured light 600. Subsets 704-1 and704-2 of signatures 704 are reflections from objects 120 and 122,respectively. Image 700 shows objects 120 and 122 in addition tosignatures 704, although in other examples, camera 116 can be configuredto capture only signatures 704 by way of a filter or sensitivitythreshold. In such examples, only signatures 704 would be visible,whereas the outlines of objects 120 and 122 would not be visible inimage 700.

Having received image 700, computing device 108 is then configured toperform block 510 as shown in FIG. 5. At block 510, computing device 108is configured to process image 700 to detect signatures that correspondto objects of interest, that is, objects defined by geometry data 136.To perform block 510, computing device 108 is thus configured todetermine which ones, if any, of signatures 704 correspond to whichportions of geometry data 136. In the present example, signatures 704-1correspond to geometry data 136, while signatures 704-2 do notcorrespond to any of geometry data 136. In other words, the detectionperformed at block 510 indicates which objects of interest are withinthe field of view of camera 116.

Having performed the detection at block 510, computing device 108 isconfigured to take different courses of action based on whether or notsignatures corresponding to objects defined by geometry data 136 weredetected in image 700. At block 515, if no signatures corresponding toobjects of interest were detected at block 510, computing device 108 isconfigured to return to block 415 of method 400. If, however (as in thepresent example performance of method 400) signatures corresponding toan object defined by geometry data 136 were detected at block 510,computing device 108 performs block 525.

At block 525, computing device 108 is configured to compare thesignatures 704 corresponding to objects of interest with geometry data136, to determine the position and orientation of those detected objectsrelative to camera 116. In the present example, computing device 108therefore compares signatures 704-1 to geometry data 136, whilesignatures 704-2 are ignored. The determined position and orientationcan be stored as data representing transformations that, when applied togeometry data 136, define a transformed version of model 200 (comparedto the “neutral” version of model 200 shown in FIG. 2) thatsubstantially matches the current position and orientation of object120. A transformed version 800 of model 200 resulting from block 525 isshown in FIG. 8. When the performance of block 525 is complete,computing device 108 is configured to return to block 415 of method 400.

The nature of the technologies used to perform blocks 510 and 525 is notparticularly limited, and generally enables computing device 108 todetermine which objects of interest are present in the field of view ofcamera 116, and what the position and orientation of those objects are.Various machine vision techniques will now occur to those skilled in theart, such as motion capture processing techniques used in filmproduction. Nonlimiting examples of such techniques are shown in thefollowing publications: U.S. Pat. No. 6,064,759; and POT PublishedPatent Application Nos. 2009/120073 and 2009/032641. Additionalinformation and alternative techniques can be found in US PublishedPatent Application Nos. 2008/0036580 and 2012/0087573, and POT PublishedPatent Application No. WO 2007/050776.

At block 415, computing device 108 is configured to generate a “canvas”image, based on the position and orientation determined at block 525,and based on image data 138. The canvas image generated at block 415 isan image to be projected by projector 104 onto projection area 112 as awhole (as shown by the dashed lines in FIG. 1), and not only onto object120 or other objects in projection area 112. In order to generate thecanvas image, computing device 108 is configured to retrieve anyportions of image data 138 that refer to objects of interest detected atblock 510, and to generate modified versions of those retrieved portionsof image data 138 by applying the same transformations that weredetermined and stored at block 525.

In the present example performance of method 400, signatures 704-1 weredetermined to correspond to object 120 (more specifically, to model 200as defined by geometry data 136) at block 510. Therefore, computingdevice 108 is configured at block 415 to retrieve any portions of imagedata 138 that contain references to model 200 in geometry data 136.Because both files in image data 138 contain references to model 200, inthe present example, images 300 and 304 are both retrieved at block 415.

Having retrieved the relevant images from image data 138, computingdevice 108 is configured to generate modified versions of images 300 and304 to match the detected position and orientation of object 120, and toposition the modified images on a digital canvas—that is, to place themodified images as portions of a single larger canvas image. Turning toFIG. 9, modified versions 900 and 904 of images 300 and 304,respectively, are shown as portions of a canvas image 912. The positionsof modified versions 900 and 904 within canvas image 912 have been setto match the positions of torso 204 and right leg 220 of model 200determined from image 700. In addition, modified versions 900 and 904have been transformed to match the orientation of model 200. The sametransformations as those determined at block 525 are applied to images300 and 304 to generate modified images 900 and 904. Suchtransformations can include skewing, scaling, rotation, cropping forpartial display, and the like.

Returning to FIG. 4, the performance of method 400 proceeds to block420, at which computing device 108 is configured to control projector104 to project canvas image 912 generated at block 415 onto projectionarea 112. The performance of block 420 therefore involves sending canvasimage 912 from computing device 108 to projector 104 via interface 132.Having received canvas image 912 from computing device 108, projector104 projects canvas image 912 onto projection area 112. As seen in FIG.10, the result of the performance of block 420 is the projection ofmodified versions 900 and 904 within canvas image 912 onto object 120such that modified versions 900 and 904 match the position andorientation of object 120, without any movement being required byprojector 104 to account for the position and orientation of object 120.

The performance of method 400 then proceeds to block 425, at whichcomputing device 108 determines whether or not to continue theprojection of images onto projection area 112. As mentioned earlier, asequence of images can be defined by image data 138, such that a videois projected onto projection area 112. For example, one or both ofimages 300 and 304 (and thus modified versions 900 and 904) can beanimated, or can be segments of video encapsulated within an arbitrarypredefined peripheral frame. In such examples, canvas image 912 isupdated (that is, the performance of block 425 is repeated) at least atthe frame rate defined by the video or animation. In other examples,image data 138 may define a length of time for which certain images areto be projected. For example, images 300 and 304 may include metadataspecifying that they are to be projected continuously for one hour. Instill other examples, system 100 may be configured to continueprojecting the same images indefinitely, until input data is received atcomputing device 108 halting the projection or altering the image datato be projected. Combinations of the above examples are alsocontemplated.

In the present example, it will be assumed that computing device 108 isconfigured to cause continuous projection of images 300 and 304(transformed as necessary, per the discussion above). Therefore, thedetermination at block 425 is affirmative, and computing device 108repeats the performance of blocks 410-420, thus projecting another“frame”. Although the same images are projected, their positions andorientations may change to account for relative movement between object120 and projector 104. FIG. 11 shows an example of a subsequentperformance of method 400, in which object 120 has moved to a differentlocation within projection area 112, and in which no modified version ofimage 304 is projected due to the right leg of object 120 (correspondingto right leg 220 of model 200) not being visible to camera 116 (notethat a modified version of image 304 is not projected onto object 122).Instead, only a modified version 1100 of image 300 is projected, at aposition and orientation matching the detected position and orientationof torso 204 (modified version 1100 is of a different size and is skewedat a different angle than modified version 900).

The frequency of repetition of blocks 410-425 is not particularlylimited. In the present example, the frequency is sufficiently high asto provide substantially real-time tracking of object 120. Thus, blocks410-425 may be performed from about sixty to about one hundred andtwenty times per second (that is, about thirty separate canvas imagesare generated per second). The above range is merely illustrative;higher and lower frame rates are also contemplated, depending on theprocessing power of computing device 108 and on the particular situationfor which system 100 is to be used.

When the performance of method 400 is repeated as discussed above, theprojection of a canvas image at block 420 and the projection ofstructured light at block 500 can be substantially simultaneous, or canalternate. For example, when projector 104 is capable of projectingvisible and invisible light simultaneously, a canvas image can beprojected at the same time as the structured light which will be used togenerate the next canvas image. In other examples, the structured lightmay be projected in between frames (that is, in between projected canvasimages), with each frame of structured light being used to generate thesubsequent canvas image.

Thus, as set out above, system 100 allows for images to be projected, asportions of a canvas image, onto specific objects in projection area112, accounting for relative motion between the objects and projector104. Although projector 104 is described above as being preferablystationary, the principles described herein can be applied to accountfor projector movement as well as object movement. For example, in FIG.12 projector 104 and camera 116 have been moved in relation toprojection area 112. As a result, object 122 obscures the right leg ofobject 120 from the view of camera 116. As a result, image 304 is notprojected.

In addition to the variations described above, additional variations tosystem 100 and method 400 are also contemplated. For example, one orboth of projector 104 and camera 116 can be replaced with multipleprojectors or multiple cameras. For example, the size of projection area112 may be such that several projectors are required to provide completeprojection coverage, and such that several cameras are required tocapture a complete image of projection area 112. In such embodiments,computing device 108 can be configured to divide canvas image 912 amongan array of projectors, and can also be configured to generate image 700as a composite of multiple images received from an array of cameras.

In another variation, two types of projectors can be provided in system100. One type can be used to project structured light 600, while theother type can be used to project canvas image 912. As mentioned in theprevious paragraph, either a single projector of each type, or multipleprojectors of each type, can be provided.

In a further variation, reflective markers can be affixed to objects ofinterest, such as object 120, in projection area 112, in order toenhance the accuracy of the determinations at blocks 510 and 525 byreducing the impact of occlusions and shadows in projection area 112.FIG. 13 shows an example of system 100 in which projector 104 projectsstructured light in the form of a full frame of infrared or other light1300, and in which object 120 carries a plurality of markers 1304 thatreflect light 1300. In this variation, the signatures described aboveare provided by reflections from markers 1304 captured by camera 116.

In a further variation to the example of FIG. 13, different types ofmarkers 604 can be fixed to different objects of interest. For example,markers having certain reflective properties can be fixed to one object,while markers having different reflective properties can be fixed toanother object. Such an implementation can reduce the computationalresources required to detect different objects and determine theposition and orientation of those objects.

In other embodiments, system 100 can be configured to control theillumination of various objects in projection area 112, rather than toproject image data onto those objects. Illumination as used hereinrefers to the projection of light without predefined spatial variationsonto an object (although there may be predefined spatial variations forprojection area 112 as a whole, due to different illumination parametersfor different objects). For example, rather than projecting image 300(which has spatial variations in colour and brightness defined by pixelarray 310) onto a given object, illuminating that object may consist ofprojecting light having the same colour, brightness, and otherattributes onto the entirety of that object. The distinction betweenimage projection and illumination will become apparent to those skilledin the art in the discussion below.

In embodiments configured to control illumination of objects rather thanimage projection, computing device 108 is configured to storeillumination data in memory 126 rather than image data 138. Turning toFIG. 14, an example of illumination data 1400 is shown. Illuminationdata 1400 can be formatted in any suitable manner, and is not limited tothe tabular format illustrated. Illumination data 1400 defines one ormore records 1404-1, 1404-2, and 1404-3, illustrated in FIG. 14 as rowsin the table. Each record 1404 defines how one or more objects withinprojection area 112 will be illuminated.

Each record 1404 in illumination data 1400 contains a reference togeometry data 136, and one or more parameters defining thecharacteristics of the light to be projected onto the object or objectscorresponding to the reference. In the example of FIG. 14, record 1404-1contains a blank or null reference to geometry data 136, record 1404-2contains a reference to head 224 of model 200, and record 1404-3contains a reference to torso 204 of model 200. A wide variety of otherreferences to geometry data 136 are contemplated. For example, a singlegiven record can refer to model 200 as a whole, or to multiple models orparts.

The null reference of record 1404-1 indicates that record 1404-1 definesthe illumination of any portion of projection area 112 that is notoccupied by an object referenced elsewhere in illumination data 1400.This includes both objects that are defined in geometry data 136 but notreferred to in records 1404, and objects that are not defined ingeometry data 136.

To illustrate the effects of the references in records 1404, referbriefly to projection area 112 as shown in FIG. 1. In that arrangementof projection area 112, according to illumination data 1400 the head andtorso of object 120 will be illuminated according to records 1404-2 and1404-3 respectively, while the remainder of projection area 112(including object 122, the background, and the legs and arms of object120) will be illuminated according to record 1404-1.

As mentioned earlier, each record 1404 also includes parameters definingillumination characteristics. For example, as shown in FIG. 14, eachrecord specifies a brightness and a colour of light to illuminate therelevant objects. The brightness can be specified in a variety offormats. In the present example, each record 1404 specifies a targetbrightness in nits (also referred to as candela per square meter, cd/m²)for the object or objects identified in that record 1404. Other units,such as foot-lamberts (fL or ft-L), can also be used. The targetbrightness. In some other examples, brightness can instead be specifiedin records 1404 as a fraction of the maximum output of projector 104.The colour can be represented in a variety of ways, including theEnglish names shown in FIG. 14, which can then be converted by processor124 into a suitable colour model (e.g. RGB, CMYK). In other examples,the colour can be represented in records 1404 in the colour model,instead of in corresponding English names. As seen in record 1404-1,when the target brightness is zero (that is, the relevant object is tobe left dark), the colour parameter can be omitted.

The brightness parameters described above are used by processor 124 tocontrol the output of projector 104 in order to achieve the specifiedtarget brightness for each object in projection area 112.

Referring now to FIG. 15, a method 400 a of illuminating objects inprojection area 112 is depicted. The performance of method 400 a will bediscussed in conjunction with its performance in system 100, although itis contemplated that method 400 a can also be performed by othersuitable systems.

Except as discussed below, the blocks of method 400 a are performed asdescribed above in connection with method 400. Thus, at block 405 thestorage of geometry data 136 is as described previously, and as notedearlier, the storage of image data 138 is replaced with the storage ofillumination data 1400. At block 410 a, the identification andpositioning of objects in projection area 112 is performed in the samemanner as described above. In addition, at block 410 a computing device108 is configured to determine the brightness of each object in theimage captured by camera 116, for instance as part of the performance ofblock 505 shown in FIG. 5. The determination of brightness can be madeby examining the captured image, by receiving data from a light meterincluded in camera 116, or other techniques that will occur to thoseskilled in the art. The brightness of a particular object or region ofan object in projection area 112 is influenced by the incident lightfrom projector 104 and any other ambient sources of light, as well ascharacteristics of the surface of the object such as (but not limitedto) reflectivity, absorption, texture, index of refraction, and so on.

At block 415 a, processor 124 is configured to generate a canvas image,which will be used by projector 104 to illuminate projection area 112.The canvas image is based on the object positions, orientations andbrightnesses determined at block 410 a, and on illumination data 1400.More specifically, based on the determinations at block 410 a, processor124 identifies the regions of the canvas image occupied by each objectreferenced in illumination data 1400. Thus, processor 124 identifies theregion of the canvas image occupied by head 224, the region occupied bytorso 204, and the region occupied by all other objects. These regionscan be generated based on the transformations applied to model 200 (orother geometry data) at block 410 a.

Having identified the relevant regions of the canvas image, processor124 is configured to fill in each region based on the parameters inillumination data 1400 and the brightness of the object in projectionarea 112 as detected at block 410 a. In the present example, processor124 is configured to fill in the region occupied by head 224 with whiteat a brightness selected to achieve the target brightness specified inrecord 1404-2. The selection of brightness for the canvas image can beperformed in a variety of ways. For example, processor 124 can beconfigured to select a brightness for the region of the canvas imageaccording to a predefined function relating target brightness withprojector output (for example, a curve plotting target brightness forvarious levels of projector output obtained by various canvas imagebrightness levels). Processor 124 can also modify the output of such afunction to account for the brightness detected at block 410 a. Forexample, a brightness selected for the canvas image can be increased ifthe detected brightness is below the target specified in illuminationdata 1400, or decreased if the detected brightness is above the targetspecified in illumination data 1400.

Similarly, processor 124 is configured to fill the region occupied bytorso 204 with red at a brightness selected to achieve the targetbrightness specified in record 1404-3, and to fill in the rest of thecanvas image with black or null lighting.

Turning to FIG. 16, a canvas image 1600 resulting from the performanceof block 415 a is shown. As seen in FIG. 16, a region 1604 of canvasimage 1600 occupied by head 224 is white in colour and has a brightnessselected to illuminate head 224 to a brightness of 200 nits. A region1608 of canvas image 1600 occupied by torso 204 is red in colour and hasa brightness selected to illuminate torso 204 to a brightness of 100nits. The remainder of canvas image 1600 is black (or, in some examples,transparent), and thus has a brightness of 0.

Referring again to FIG. 15, after generation of canvas image 1600, atblock 420 a processor 124 is configured to control projector 104 toproject canvas image 1600 onto projection area. The results of theperformance of block 420 a are shown in FIG. 17. As seen in FIG. 17,canvas image 1600 is projected onto projection area 112 by projector 104(under the control of computing device 108), and the portions discussedin connection with FIG. 16 coincide with the location of the objectscorresponding to the geometry data reference in illumination data 1400.

Processor 124 is configured, following the performance of block 420 a,to determine at block 425 a whether further projection is necessary, asdiscussed above in connection with block 425.

Still other variations to the above systems and methods will also occurto those skilled in the art.

Those skilled in the art will appreciate that in some embodiments, thefunctionality of computing device 108 executing application 134 can beimplemented using pre-programmed hardware or firmware elements (e.g.,application specific integrated circuits (ASICs), electrically erasableprogrammable read-only memories (EEPROMs), etc.), or other relatedcomponents.

Persons skilled in the art will appreciate that there are yet morealternative implementations and modifications possible for implementingthe embodiments, and that the above implementations and examples areonly illustrations of one or more embodiments. The scope, therefore, isonly to be limited by the claims appended hereto.

We claim:
 1. A system for illuminating objects in a projection area,comprising: a computing device storing: geometry data defining a digitalmodel of an object; and illumination data containing a record having (i)one or more parameters defining characteristics of light to be projectedonto the object, and (ii) a reference to the digital model; a lightsource connected to the computing device and configured to projectstructured light onto the projection area; a camera connected to thecomputing device and configured to capture an image of the projectionarea during the projection of structured light; the computing deviceconfigured to receive the captured image from the camera and todetermine a position and orientation of the object in the projectionarea by comparing the geometry data to the captured image; the computingdevice further configured to generate a canvas image including a regionmatching the determined position and orientation of the object, theregion having a fill defined by the one or more parameters; thecomputing device further configured to transmit the canvas image to aprojector, for projection onto the projection area, whereby a portion ofthe object corresponding to the reference in the image data isilluminated according to the one or more parameters.
 2. The system ofclaim 1, the computing device configured to determine a position of theobject by detecting signatures in the captured image corresponding tothe digital model, and comparing the signatures to the geometry data. 3.The system of claim 2, the computing device configured to applytransformations to the digital model such that the transformed digitalmodel matches the signatures; the computing device further configured togenerate the region of the canvas image based on at least one of thetransformations.
 4. The system of claim 1, wherein the light source is acomponent of the projector.
 5. The system of claim 1, wherein theprojection area contains at least one of a plurality of objects; andwherein the geometry data defines a plurality of digital modelscorresponding to the plurality of objects; the computing deviceconfigured to detect signatures corresponding to at least one of theplurality of digital models in the captured image.
 6. The system ofclaim 1, wherein the one or more parameters include a target brightnessparameter and a colour parameter.
 7. The system of claim 1, wherein thegeometry data includes identifiers of parts of the digital model; andwherein the reference in the illumination data consists of at least oneof the identifiers.
 8. The system of claim 6, further comprisingdetecting a brightness of the object, and selecting a fill brightnessfor the region based on the target brightness and the detectedbrightness.
 9. The system of claim 7, wherein the illumination dataincludes an additional record having a null reference to the geometrydata.
 10. A computing device for use in a system for illuminatingobjects in a projection area, the computing device comprising: a memorystoring: geometry data defining a digital model of an object; andillumination data containing a record having (i) one or more parametersdefining characteristics of light to be projected onto the object, and(ii) a reference to the digital model; a data interface configured tocommunicate with a light source, a projector and a camera; and aprocessor interconnected with the memory and the data interface, theprocessor configured to: control the light source to project structuredlight onto the projection area; control the camera to capture an imageof the projection area during the projection of structured light;receive the captured image from the camera, and determine a position andorientation of the object in the projection area by comparing thegeometry data to the captured image; generate a canvas image including aregion matching the determined position and orientation of the object,the region having a fill defined by the one or more parameters; andtransmit the canvas image to a projector, for projection onto theprojection area, whereby a portion of the object corresponding to thereference in the illumination data is illuminated according to the oneor more parameters.
 11. The computing device of claim 10, the processorconfigured to determine a position of the object by detecting signaturesin the captured image corresponding to the digital model, and comparingthe signatures to the geometry data.
 12. The computing device of claim11, the processor configured to apply transformations to the digitalmodel such that the transformed digital model matches the signatures;the processor further configured to generate the region of the canvasimage based on at least one of the transformations.
 13. The computingdevice of claim 10, wherein the light source is a component of theprojector.
 14. The computing device of claim 10, wherein the projectionarea contains at least one of a plurality of objects; and wherein thegeometry data defines a plurality of digital models corresponding to theplurality of objects; the processor configured to detect signaturescorresponding to at least one of the plurality of digital models in thecaptured image.
 15. The computing device of claim 10, wherein the one ormore parameters include a target brightness parameter and a colourparameter.
 16. The computing device of claim 10, wherein the geometrydata includes identifiers of parts of the digital model; and wherein thereference in the illumination data consists of at least one of theidentifiers.
 17. The computing device system of claim 15, the processorfurther configured to detect a brightness of the object, and select afill brightness for the region based on the target brightness and thedetected brightness.
 18. The computing device of claim 16, wherein atleast one of the one or more parameters is dynamically variable.
 19. Amethod of illuminating objects in a projection area, comprising:storing, in a memory of a computing device: geometry data defining adigital model of an object; and illumination data containing a recordhaving (i) one or more parameters defining characteristics of light tobe projected onto the object, and (ii) a reference to the digital model;controlling a light source connected to the computing device to projectstructured light onto the projection area; controlling a cameraconnected to the computing device to capture an image of the projectionarea during the projection of structured light; receiving the capturedimage at the computing device from the camera and determining a positionand orientation of the object in the projection area by comparing thegeometry data to the captured image; generating a canvas image at thecomputing device, including a region marching the determined positionand orientation of the object, the region having a fill defined by theone or more parameters; transmitting the canvas image to a projectorconnected to the computing device, for projection onto the projectionarea, whereby a portion of the object corresponding to the reference inthe illumination data is illuminated according to the one or moreparameters.
 20. A non-transitory computer readable medium storing aplurality of computer readable instructions executable by a processor ofa computing device, for causing the processor to perform a method ofilluminating objects in a projection area, the method comprising:storing, in a memory of a computing device: geometry data defining adigital model of an object; and illumination data containing a recordhaving (i) one or more parameters defining characteristics of light tobe projected onto the object, and (ii) a reference to the digital model;controlling a light source connected to the computing device to projectstructured light onto the projection area; controlling a cameraconnected to the computing device to capture an image of the projectionarea during the projection of structured light; receiving the capturedimage at the computing device from the camera and determining a positionand orientation of the object in the projection area by comparing thegeometry data to the captured image; generating a canvas image at thecomputing device, including a region matching the determined positionand orientation of the object, the region having a fill defined by theone or more parameters; transmitting the canvas image to a projectorconnected to the computing device, for projection onto the projectionarea, whereby a portion of the object corresponding to the reference inthe illumination data is illuminated according to the one or moreparameters.